1. Field of the Invention
The present invention relates to image processing for correcting a blur of an image.
2. Description of the Related Art
Known image recovery algorithms recover an image captured by an image capture apparatus from degradation due to, for example, out of focus, aberration, or camera shake. For example, one method describes the characteristic of image degradation by a point spread function (PSF) and recovers a non-degraded image based on the PSF. Japanese Patent Laid-Open No. 62-127976, for example, discloses, as such an image recovery algorithm, a method of correcting a blur of an image by filter processing with a characteristic inverse to a PSF.
Let (x,y) be position coordinates within an image, o(x,y) be a non-degraded image (to be referred to as a subject image hereinafter), z(x,y) be an image suffering from degradation (to be referred to as a degraded image hereinafter), and p(x,y) be the PSF describing information on point images spreading due to a blur. These three characteristics z(x,y), o(x,y), and p(x,y) satisfy:z(x,y)=o(x,y)*p(x,y)  (1)where * represents convolution calculation.
Equation (1) can be rewritten as an integral equation:z(x,y)=∫∫o(x,y)p(x−x′,y−y′)dx′dy′  (2)
Equation (2) is Fourier-transformed into the domain of a spatial frequency (u,v) as:Z(u,v)=O(u,v)·P(u,v)  (3)where Z(u,v) is the spectrum of z(x,y), O(u,v) is the spectrum of o(x,y), and P(u,v) is the spectrum of p(x,y).
Note that P(u,v) is a modulation transfer function (MTF) as the absolute value of an optical transfer function (OTF) which is the two-dimensional Fourier transform of the PSF.
As long as the PSF p(x,y) can be known in some way in addition to the degraded image z(x,y), a spectrum O(u,v) of the subject image can be calculated by calculating the spectra of z(x,y) and p(x,y) and using an equation obtained by rewriting equation (3) as:O(u,v)=Z(u,v)/P(u,v)  (4)where 1/P(u,v) is called the inverse filter.
There often exists a frequency at which the MTF value of degradation is zero. That the MTF value is zero means that there exists a frequency component which is not transferred (whose information is lost) due to degradation. When there exists a frequency at which the MTF value is zero, the subject image cannot be perfectly recovered. Hence, there often exists a frequency at which the inverse filter of an MTF has an infinite coefficient, and the subject image has an indefinite spectrum value at that frequency.
Under the circumstances, to prevent the coefficient of the inverse filter from becoming infinite, a Wiener filter described by:P(u,v)/{|P(u,v)|2+c}  (5)is often used for image recovery, where c is a very small constant.
An inverse filter or a Wiener filter will be referred to as a “recovery filter” hereinafter. The coefficient of a recovery filter is inversely proportional to the MTF and therefore increases as the MTF comes close to zero. That is, recovery filters for frequencies with great degradation have very high coefficients. Thus, if a PSF for use in calculation of a recovery filter is different from that of an actual blurred image, filter processing amplifies their difference though it may be small. In other words, an accurate PSF is desirably acquired to recover an unblurred subject image from a degraded image.
A PSF is well known to change depending on the image height, the zoom ratio, the subject distance, and the stop. In view of this, one proposed method calculates a PSF in accordance with these items of image capture information and feeds them back to a recovery process. For example, Japanese Patent Laid-Open No. 4-088765 estimates a PSF corresponding to the subject distance and uses it for recovery from image degradation. Japanese Patent Laid-Open No. 2004-205802 pays attention to the fact that a change in luminance of a subject in the shutter open duration is large when a flash is used, and the PSF at this time is different from that in the shutter open duration when no flash is used (a change in luminance is less). In this case, a recovery process is performed by correcting the PSF when a flash is used.
However, the PSF differs depending on the type of light source in image capture even when the image height, the zoom ratio, the subject distance, and the stop are the same and a change in luminance in the shutter open duration is uniform. For example, the PSF under light source A is different from that under a warm white fluorescent lamp.
FIG. 1A is a graph showing PSFs. Note that a plot of the PSF values has a three-dimensional shape because point images spread in the vertical and horizontal directions of the image. However, for the sake of easy explanation, the PSF values are two-dimensionally plotted by paying attention to only a one-dimensional direction of the image in FIG. 1A.
FIG. 1A shows PSFs when the image height, the zoom ratio, and the subject distance are the same and a change in luminance in the shutter open duration is uniform under two different light sources. However, the two light sources generate different point image spreading characteristics, that is, different PSFs.
FIG. 1B is a graph showing the MTFs of the Fourier transforms of the PSFs shown in FIG. 1A. In FIG. 1B, the abscissa indicates the spatial frequency, the origin indicates zero frequency, and the frequency increases in a direction away from the origin. The MTFs shown in FIG. 1B have only a small difference between the two light sources except in the vicinities of the two ends of the graph corresponding to high-frequency components.
FIG. 1C is a graph showing the reciprocals of the MTFs. That is, FIG. 1C is a graph describing the spatial frequency characteristics of recovery filters. As can be seen from FIG. 1C, high-frequency components have only a small difference between the MTFs shown in FIG. 1B, but they have a large difference between the reciprocals of the MTFs.
It is therefore difficult to obtain a satisfactory recovery result even by using recovery filters created using the PSFs of different light sources. That is, if an image captured under light source 1 undergoes a recovery process using a recovery filter created based on the PSF of light source 2, high-frequency components are insufficiently recovered. As a result, the image after the recovery process still remains blurred. In contrast, if an image captured under light source 2 undergoes a recovery process using a recovery filter created based on the PSF of light source 1, high-frequency components are excessively recovered. As a result, image degradations such as stains or ringing occur. Depending on the circumstances involved, the image after recovery may even be poorer in quality than that before recovery.
FIGS. 2A to 2D are views for explaining the negative effects in a recovery process. FIG. 2A shows a subject image, and FIG. 2B shows an image generated by blurring the subject image based on the PSF of a certain light source (light source 3).
FIG. 2C shows an image (to be referred to as a recovered image hereinafter) obtained by applying a recovery filter optimum for light source 3, created based on the reciprocal of an MTF associated with light source 3, to the image shown in FIG. 2B. A satisfactory recovery result is obtained in the recovered image shown in FIG. 2C.
In contrast, FIG. 2D shows a recovered image obtained by applying a recovery filter optimum for a light source (light source 4) different from light source 3 to the image shown in FIG. 2B. The recovered image shown in FIG. 2D suffers from ringing on its edge portions. Hence, the image after recovery (FIG. 2D) is poorer in quality than the image before recovery (FIG. 2B).